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ABSTRACT • 

The study was designed to examine^ the similarity of 
response options generated by different item writers using a 
systematic approach to item writing. The similarity of response 

student responses for the same^item stems presented in an 
format was also examined. A^ nojf- systematic! (subject matter 
approach and a systematic '(algorithmic) approach were the 
item writing' employed. Results indicated that neither * 



options to 
open-ended, 
expertise) 
methods of 



approach resulted in ve*ry similar response options bein^ generated by 
writers. % Few response options produced by either approach 

\ corresponded with the incorrect* student' responses*, The variation in 
incorrect student responses illustrated that the use of multiple ^ 
choice items to test basic mathematics operations ^would not "account 
for all student errors. * It is probable that multiple choice test 

•scores would be inflated over open-ended format test scores utilizing 
the same items. The study highlighted some potential problems in 
attempting to construct unbiased multiple choice tests. (DWH) 
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The Technical Quality of Test? Items Generated Using a Systematic 

* ■» 
Approach to Item Writing 

• ' ' v 

The intrusion of criterion-referenced testing into the educational 
^scene has led to several interesting ittd potentially useful developments. 
One of these developments fcas resulted from the call for greater 

specificity of item writing rules and operations if test scores are to 

\ 

1 

become ipcreasingly meaningful. Bormuth (1970) was one of the first 
tt> emphasize the need for s # uch increased 

Bormuth, results obtained from tests made by traditional methods cannot 
be used as evidence in deciding issue^otVpublic policy or instructional 
theory- because "there is no way to refute or confirfh the results of a study 

f 4 

in which items made by traditional methods were used. The reason for 
these problem is that items made in the traditional manner are derived 
through private intuition of the item writer rather than through a set of 
operations open to public inspection " (Bormuth, 1970, p. 7; -emphasis ours). 

In Response to. this identified problem Bormuth devised an approach 
to it$m writing which he termed "item transformations." The main purpose 
of /the approach was to increase the similarity of achievement test items 
written by t different item writers to assess student acquisition of the same 
objectives. Subsequently, a number o£ educator^ and testing experts developed 
other Systematic, explicit approaches in order to minimize so-called "item 
writer bias." Hively et al/s, (1973) .item forms, Durn^n and Scandura's (1973) 
algorithmic approach, Popham's (1975) amplified objectives, BeiVs (1978) 
mapping sentences, and Tiem^nn and Markle f s (1978) £onc~ept-based testing 
represent five of the most potentially useful approaches ^ In a recent review 
of^systematic item writing approaches Roid and Haladyna (1980) endorse the 

/ , o 



*cont;inued and increased use of these approaches while contending that 
f, (m)any of. the problems of item-writer biap may £e avdided by employing one 
of several 'domain-based it;em-gQ^eratlng approaches 11 (p. 309). 

Despite the ma§ni,tude*of the problem identified by Bormuth^ and the 
endorsement by Raid' and Haladyna of systematic itefa writing approaches 
as potential solutions to fche problem, relatively lijtle research has been 
conducted to examine the validity^of the vendors emenfc. The research conducted 
to date has'* deployed a traditional index of item difficulty a^ the dependent 
variable ^Haladyna and Roid, 1978). That is, if item^ge^eraUed by different 
item writers are associated with similar item difficulties, -item-writer bilas 
is said to have been minimized or eliminated. - * 

The use of an index of item difficulty as an indicator of item-writer 
bias is problematic for at least twp.reasons. Firs-t, if >ias is defined 
in terms of the deviation from a "true" difficulty, theh similarity of item 
difficulties may or may not indicate the'-elimination of bias: Perhaps both 
of the item writers employed in one of .the Haladynaand Roid studies were equal 
ly and systematically biased. >[f so, little* variation in , the difficulty of 
the -resulting,*! terns, would be a likely result. Second, a/comparison of 'the 

' * ' 1 ' ( ' , ' 

items theSaselves (in addition to pttidents 1 responses to the if ems) .seems 
necessary if item-writer bias is to be adequately examined and, ultimately, 
explained . - # , , 

The issue of . item-writer, bias for \^ultiple.-choide tests includes two. 
facets. 1 First, item stems derived by different item writers from ia given 
content area should b$ similar as well as representative of the content 
tested. Second, response optiotis should , be replicable across* item writers 
while* also mirroring actual student errors.; The present study -focuses on 
the latter of these two aspects -of item writer bias. 4 



More specifically, the purpose* of the study were to examine 1) the 
similarity 'of response, options generat^by different item writers using -a 
systematic approach to item .writing, 2) -the similarity of response options 
generated by different item writers who were subject matter specialists and 
had no knowledge of systematic approaches to item writing, and 3) the 
similarity of the response options generated by both aforementioned groups 
of item writers to student responses for the same item stems presented in _ 

• V v .- 

completion, rather than selection, format. 

Samples and Procedures 

Three samples were included in the study. Two samples consisted of 
» ■ 

item writers; a third was composed of sixth grade pupils. , The two samples 
of item writers differed both in terms of their educational background and 
in the instructions they received relative to the generation of response op- 
tions. • ' 

Sample '% consisted 'of four certified secondary- level teachers holding 
Masters degree's in mathematics education. These teachers were instructed to 
examine twenty arithmetic exercises (five problems each for addition, sub- 
traction, multiplication, and 'division of whole numbers). Based on .their ^ 
knowledge of the way in which sixth grade students learn to work these types 
of problems, these item writers were asked to generate response options that 
would' correspond to errors such students would be likely to' make. Three 
incorrect response options were to be generated. for each exercise resulting 
in a total <?f 60 incorrect response options. 

Sample B was composed of four undergraduate students enrolled in a 
test construction course during the next-to-last semester of their senior 
year. AU of these students were majoring in early childhood, elementary, 
, or special education. All had previously completed one course in mathe- 
matics education. 



One segment of the test construction cburse was devoted to systematic 

approaches to item writing. During this Segment, one ninety-minute class 

? * 
period was spent on Durnin and Scandura 1 s,. £1973) algorithmic approach. 

( \ " * 

Briefly, the algorithmic approacfh is ba^ed on/ the assumption that students 

• . ' ' - 

use rules to solve problems. *By taking account of the rules — professes 

v S 
• } 

and decision points — that students use to solve problems, a £lo\* chart 

* * * * 

• can be diagrammed for a given type or category of problems*. 

* Four flow charts diagramming procedures fdr solving addition-, subtrac- 

i 

tion, .multiplication and division problems for whole numbers had.baen con- 
structed by th^ instructor of the course. '(Each flow chart hp! previously 
been validated by a teacheryf eview process involving at_ least three teachers,) 
The undergraduate s^SiWts were trained to rea^ the flow' charts. In addi- 
tion, students^wer^ told that incorrect response options for math problems 
could be generated by utilizing the flow charts. In the derivation of 

options', these stydents were instructed to make -either a process error (an 

r 

error in performing a step or specified activity) or a decision point 

error 0 (an error in selecting the appropriate route through the flow chart). 

The erroneous process or path was to -be followed until the algorithm^ was 
* t ♦ 

completed, resulting in an incorrect response. Thus, for each pass through 
the flow chart, -a unique error was. to be committed. As in the case of the , 

v • • *. . 

item writers s in Sample A, those in Sample B were instructed to generate 

60 response options for the 20 arithmetic exercises. 

* * * • * 

Sample C £onsist;ed of I3l sixth grade' pupils enrolled in one of si^: 

^mathematics classes. Three different teachers, each teaching two cl^te^es , 

instructed the six classes. The same texttpok was used in all of the 

* 

classes. ■ A completion test consisting of only the 20 arithmetic exercises 

was prepared and distributed. to the students. The test directions asked 



; 1 



9 

ERIC 



V 

ERIC 



the students 'to wo'rk each problem and write the answer on the tes.t itself. 
No time limits were imposed for completion of the test. 

Findings Concerning Similarity of Res ponse Options Generated by Each Approach 

Analyses of" the response options generated by the item writers using 
the algorithmic approach , and . those having subject'matter expertise suggest 
that neither approach, independently, yielded very similar distracters. ■ 
Considering the total number of response options generated by ^oth samples 
of item writers, 71 per cent of those generated by the subject matter 
experts differed from o$e another. Similarly, 8Q per cent of the responses 
generated by the item writers employing the algorithmic approach were 
dissimilar. *The number and percentages of different response options de- 
veloped by 'each approach "are reported for each item and all items in Table 1. 



Insert Table 1 about here 



* 

' Table 1 provides an indication of the variation in generated response 
options for each approach. However, even if all item writers duplicated 
.each other's distracters^, the percentage qf different responses would be- 
25 per cent (indicating each. of the four writer's had derived the same three 
options). Table 2 further explicates Table 1 by reporting how miny item 
writers for each approach, separately,' derived the^ same response option.- 



Insert Table 2 about here 



Despite classification b«y Roid and Haladyna (198oj as one of the more 
objective item writing approaches, on one plane the algorithmic approach is 
no less "biased" than more traditional approaches to item. writing. In.' fact, 
as Tables 1 and 2 indicate, the algorithmic approach tended to result in a _ 
greater variety of response options than the non-systematic 'approach. 



When examined from another perspective^however , the algorithmic 
approach may be well be viewed as relatively "unbiased". If indeed the 
algorithmic approach does account for rules governing the thought proc- 
esses, one might expect response options generated by this approach* to vary y 

"according to the number of . processes and decision points present in the 

V ^ 

correct response algorithm. Thajt if, the more processes, and decision 

points, the greater the^chance for student error, and the greater the pumber * 

of response pptions^ that can be generated. v 

To investigate this possibility, the percentage of dissimilar, response ^ 

options generated by the algorithmically-based .item writers was correlated 

with the number of procedures and decision points present in the correct 

response algorithm. The resultant correlation was 0.51 (p <.05"J. This 

finding suggests that the algorithmic approach to response^option deriva- 

tion is^ sensitive to .the complexity of the procedure being tested. 

» m 

Further analyses resulted in a correlation* of 0.55 between the' number 
of dissimilar response options generated by the two samples of item writers. 
Additionally, the correlation betwfeen number of dissimilar response options 
written by the subject matter specialists and the number* of processes and 
decision points in the correct*response Algorithms was 0.47. These findings 
suggest that subject toatter specialists may; be intuitively following rules 
in deriving their response options. " r 

Findings Concerning Similarity of Response Options Generated by Both Approaches 

In addition to a great djEj^, of "within-apprpach" variability among r$- * 
sponse options, a great deal^^r cross-approach variability was $lso - apparent. 
Table 3 reports the number of response options generated for each of the 20 
exercises by writers for both approaches and the number of writers for each 
approach deriving the same option (s) common to both approaches. 
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Table 3 illustrates that "there is little .commonality "between response 
options generated by the two apprqaches. -A closer examination .of the/factual 
response options indicates that this finding may be attributable, in parti . 
to inherent differences .4n the. two approaches. .In specific, the algorithmic \ 
approach discourages individuality in. errors and 'encourages continued' com- • 
mission- of an error once it is 'committed. Thus; for example, in a three- ■ 
digit subtraction problem with .borf owing, once item writers using the 
algorithmic approach commit a borrowing error, the same error should be 
committed throughout the problem. Whereas, subject flatter specialists may 
employ their own judgment as to whether or not an error will be continually 
committed*. 

Findings Concerning Similarity Between Generate d Response Options and Supplied 
Responses 

A compares?* of response options generated by writers using both 
approaches with responses supplied by the sixth-grade students indicates that 
relatively few of the distracters produced by either sample of item writers 
. correspond with incorrect responses made by the students. Of all incorrect 
responses made to the 20*- exerciser,. 13 per cent were ' identical to response 
options generated "by the subject' matter specialists and 20 per cent were 
the Irame as those produce^ by the item writers using the algorithmic approach. 
Table 4 reports the number of incorrect responses made by the students to 
each item as well as the number and per cent of student-supplied responses 
corresponding to the response options generated by the two samples of item 

♦ 

writers. 



Insert Table 4 about: here 
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The algorithmically-derived Tiistracters matched more of the incorrect 
student responses than options derived by the subject matter experts for 
•13 of the 20 items. * However, neither approach would have resulted in * * 
multiple-choice items in which response options matched the majority^ of 
incorrect student? responses. Furthermore, 'the* analyses summarized in 
Table 4 utilized all response options generated by«item writers (i.e. ^10, 
11 or 12 response options). MUltiple-choice ijtems typically allow onl^ 
four or five options. t 

* Interestingly,' the ^response options generated by the greatest number 

of item-writers were frequently not the responses which most often matched t 
the correct student responses or the most frequently occurring incorrect 

.student responses. Consider, for example, Item 8 which wa£ answered incor- 
rectly by, 38 students. Nineteen '(or 50 per cent) of these incorrect re- 
sponses matched distracters generated by the subject matter specialists 
(see^Table 4). However, the three response opfcions. most frequently generated 

by the subject-matter specialists corresponded with only two of th^ students 1 

*\ % * . .? 

incorrect responses. In contrast, response options generated by a single 

« • '■ * - t ^ t 

subject matter specialist corresponded with the infcferjrect responses made 

4 * 

' , * I 

by eight students. . * 

This phenomenon is especially problematic with respect to the g^nera- 
tion of multiple- choice items! If,* in constructing, Item, 8, one had selected 

in addition to the correct answer, rthe three response fcptiQns .generated -most 

In- 
frequently by writers, only 95 (73%) o£ the student responses would have , 

matched the multiple- choice options. If, on the other hand, one had 

t 

incorporated into Item 8T, options less frequently supplied by writers, 110 

1 \ 
(84%) student responses would haye matched the options. m • ^ — 



One major contributor to the lack of congruence between the' response ^ ^ 
options generated by the item writers 'and th<* responses supplied by the - 
•students .is. lack of consistency- /mong the incorrect responses »of students,. 
Table 5 reports the number of students responding incorrectly to each! item- 
and tihet number and percentage of different responses to each item. • A dis- 
tinction is drawn, between the number of students missing the item (N Incorrect) 
and the number of students who attempted the item and missed it (N Answering). 
In-a test situation, ah unanswered question would most likely.be marked 
incorrect*. In an analysis 'of responses, however, questions left unanswered 
by different, students do not necessarily represent the same response or 
terror Whereas one student may have overlooked the item, another may not 
' have been able to compute the answer. 



Insert Table 5 about here. 



The analyses summarised by Table 5 show that among students who answer 
addition, subtraction, multiplication and division problems incorrectly, ' 
.few arrive'at the same inporrect^answe^r . An attempt to analyze the students'; 
incorreot responses across items uncovered only one student whose incorrect 
responses exhibited a pattern, and for that student, only four of five in- 
correct responses followed the pattern. Further analyses uncovered little 
relationship between the number of different -incorrect student responses and 
the number of processes and decision points in a. correct response algorithm 
(r = 0.18, p /.05). It appears from these preliminary ' analyses that most 

c . " / 

student errors in responding ' to , this type of test problem are random; that , 
'is, they follow no systematic pattern. Item writers are unlikely to generate 
random error response^ options , thus in part, explaining the lack of congru- 
ence between student responses and the generated response options* Eurther- 
more,, student errors may be compounded, thus partially explaining why student 



10, • 

< 

responses were So poorly matched to generated .response options, especially 
algorithmically ^derived options. 

Discussion \ 

Recent item writing technology has emerged, at least in part, as a 
response to lack of replication among item writers. The purposes of this 
study were 1) to examine whether or not different item-writers would generate 
similar multiple-chffice response options, and 2) * to examine whether or not 
• the response options were similar to actual student responses to the £ame 
item .stems presented in open-ended format. 

In answering the first purpose, two different "methods of item writing 
were employed - a non-systematic (subject matter expertise) approach and a 
systematic (algorithmic) approach. Findings indicated t]hat neither approach 
resulted i^n very similar response options being generated by writers. Although 
in one respect this may be vievfed as evidence of item' writer bias, frcyn another 



pdint of view, this finding may be viewed as evidence of minimization of item 

rent : 



writer bias in that the number of different response options is related to 



the underlying difficulty of the item. 

Few options were generated by both approaches. By nature, the approaches'" ^ 
are very different, therefore, they 'would not be expected to produce the s*me 
response options. However, a closer inspection^ the actual options in~ 
"dicatasthat the restriction in the algorithmic approach to one error per pass 
contrasts with the employment of judgment by subject matter experts. J 1 * 

With regard to the sectpnd purpose of the study, student, responses to 
open-ended items were compared to the responses generated by the two samples 

of item*writers. Few response options produced by either approach corre- 

* 

sponded with the incorrect student responses. Item-writer response options 
» which -did correspond to the student responses, were rarely the most frequently m 
generated tii^trac't^rs . .One reason fjr the, lack of congTtr^nce between item- 
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writer generated options and student-responses was the variety of incorrect 

student responses. . v 

. - • \ ' 

Such variation in incorrect student responses makes it apparent that the 
tise of multiple-choice ipems to test basic mathematics operations wilf not 
account for all student errors". Therefore, it is 4 likely that, scores on 
multiple-choice tests wou,ld be, inflated oyer scores on tests utilizing the 
sam items in open-ended format. Students Wo might work problems incorrectly 
in completion format, would in a multiple-choice test be signalled to Rework 

• * 

the problem (or guess) by' the absence of an option matching their incorrect 

f 

work* ^ 

•This final assumption is yet to^be tested' empirically . However, the 

present study provides initial 4 data leading to such a conclusion, and high- 

m ' ' v 

lights some of the ptoblems likely to ,be encountered in attempting to, con- 

struct s,, unbiased" multiple- choice tests. % 
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J * Table 1 

Number t anfl Percentage of Different Response, Options 
Generated by Non-Systematic and Systematic Approaches - 
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H— 




t 


Approach to 


■ -v : ' 
the Generation of Response Options 








Subiect Matter Expertise • 


Algorithmic Approach 


ITEM 


*t — 

? 


(N different/N written) 


percent 
different 


(N differeht/N written) 


different 


1 


1 


( 7/12) 


58% 


(10/12) 


83% 


2 


\ 


( 8/12) 


1 66% 


r (io/i2) 


83% 


3 




'( 7/12) 


58% 


' (11/12) 


92% 


,4 




( 7/12) 


58% 


( 8/12) • • - 


66% 


5 , 




( $12) 


66% 


( 9/12) 


75% 


6 


/ 


( 6/11) 


55% 


( 6/12) 


*50% 


7 


<? 


k 8/12) 


66%. 


( 8/12) 


66% 


8 


% 


( 6/12) 


"50% 


( 8/12) 


66% ' 


9 




( 8/12) 


op 5 * 


f 7/151 , 




10 




( 8/12) 


y 66% 


• ( 9/12). * ' 


' 75% 


11 

12 




( 9/12) 
(10/H) 


75% 

91% 


(10/12)* . ' 
(11/12) ■ * 


83% 
92% 


13 




( 7/12) 


58% 


( "9/12) 


75% * 


14 




(10/12) 


83% 


(11/12)' 


52% 

* 


15 




£ (10/12) 


83% 


(10/12) - 


83%' 


16 . 




k 9/10) 


90% 


( 9/10) 


75% 


17 


i 


(10/12) 


■ 83*% 


• (11/12) • rs 


92% 


,18 




^y. ( 9/10) 


90*% 


(11/12) 


.92% ' - 


19 




( 8/12) 


66% 


, (11/12) / 


92% 


! 20 




(11/12) 


92% 


(12/12) J 


100% 


TOTAL 




(166/234) 


71% 


(191/240) 


80% ,' ■ 




' " , v Table 2 * 

^Number Each of Different Response Options Generated ^ 

by Non-Systematic and Systematic Approaches* %. 

^ 

Aooroac.h to Generation of Response Options 


* 

Item 


Subject Matter 'Expertise 
(Number of Different Writers Deriving Option) 


AigonLniuio' A^jproacn 
(Number of Different Response Options Derived) 


1 2 3 4 5 6 7. y8 9 10 11 ( 12 


1 2 3 4 5 . 6 7 8. 9 id) 11 12 


1 


.4 2 2 V 1 11 1 • -J ' . 


^ ' 7 ' . ; 

2* 2 ' 1 1 l* .1- " 1 ' 1 1 ' i 




5 — T Xs i 

3-311', 1, 1 1 'l . 1 


22 1111111 '1 


3- 


3^21111; ' 


2111-11111 1 1 - . 




3 3 2 .1 1 1 1 


'.4 2 111 111. w • ' 


5 


% 3 2 2 1 "1 1 *1 -1 . 


2 - fi .2 2 1' 1" 1 1 .1 .1 ' 


6 


4 3 • 1 1 1 1 


c ~" — ' — arffSI •" 

4 3 2 1 . { 1 1 9r. 


7 


^ ; . ■ = 1 

3 3./ 1 1 • 1 1 1 


2 2 2 2 1 1- !• 1 


• 8 * 


3 3-3111 


3 2 21 1 1 11 ^ 


9' 


4 2 1 1" 1 1 1% 1 


; 1 * Mf " 

3 2 2 1 1 1 1 


10 


3 3 1 1*1 1 1-1 


. 2221111^l l' 


11 


321* -11111 * s 


'2 2 1 1 1 1 1 1- 1/1 


12 ' - * 


2-2 11l'llll 1 

• 


2 *1 1 1 1 1 1.1 1 1 ' '1. . 


13 


3 2 2 .2 4 1 1, , 


* 

321111111 


14 ' 


2211111. 11 1 


21111lfll.l 1. 


15 


. p , 

2 2 . a i i.i i • l i i * 


2 2 11111111 


16 


2 . ",.1 1 1 ^ 1 1 - 1 1 


'222111111 


17 ' 


22-1 '111111 1 


211111111 1 1 

« L . ■ ' ' 


18 


2 1 1*1 1.1.1 1 1 * • 


2 111,11^111 1 1 


19 


3 3 111 I'll* 


2-11111111 1 1 


20'' - 


.2 1 1. 1 1 1 1 1 1 1 1 

• .1 


11111-1111 1 1 1 



q ^Jiijmbered response options do not necessarily correspond across approaches. 
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Table 3 



Response Options Generated by Writers of .Both Approaches 
and Number of Writer's for Each Approach Generating Op-tio 



Item 


Options Generated 
by Writers of B 
Both Approaches 


Number of 
' Subject Matter , 
Experts Generating Option 

/ ; 1 ~ 


■ .!} k > 

NumDer oi * 
Algorithmic Approach 
Writers. Generating Option v 


1 


1 

2 


1 

'4 


4 m 

• •2 
1 


2 


1 

2 


3 
1 


9 * 

l 


3 


* t 1 

■ 2 
'3 


3 


.1 ^ 

1 
1 

l 


4 

* 


1 

2 . 
3 


3 
2 
1 


l 

9 
Z. 


5 


1 

2 
3 


3 
2 
1 


-t 


6 


1 , 

2 

3 


* 1 
4 
3 


3 
4 


+1 


1 
2 
3 

* 4 


. 3. 

3 . 

l 

l 


1 

Z 

2 
1 


8 


1 

2 
3 
4 
5 


l ' 
l 

3 1 

3 

3 


1 

3 

9 i* 
Z 

2 
1 


9 


3 
4 
5 


1 

1 . ' 

1 

4 

. 1 


1 

J 

2 
2 

1 


10 


1 

2 

■2 
J 

' v 4 - 


1 
3 
3 
I 


2 
1 
1 
2 


11 


1 

~^ ' 2 
3 


1 
3 
2 


1 

1 '- ■ 


12 

k 


1 

• 2 
3 , 


1 
1 
1 


1 
1 

J. 

i 

3 


13 
14 


1 
1 

z 


3 
2 
1 


1 

2 


15 


1 

2 
3 


1 

1 ' 
2 


1 
1 

2 


16' 

L 


1 

2 
3 
4 


1 

1 • . 

, 1 

' 2 


1 
2 
1 
2 
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Table 3* (continued) 



Response Options* Generated by Writers of Both Approaches 
atid Number of Vripers for Each Approach Generating Option 



Item 


Options Generated 
J>y Writers of 
Both Approaches 


Number o£ 
Subject Matter 
Experts Generating' Option* 


f numDer or 

Algorithmic Approach 
— , Writers Generating Option 


17 


0 






18 


2 * 


i 
i 


1 

1 _ 


19 


1 


3 


2 


20 


0 


° 






1 — * 




i 

* 












* 


\ ■ 

* • 






- \ 

PS 


* - 


» 


- 


>. 

I 


t 

« 1 
i 


♦ 




• 


t 




• 




< 


f 


V 


\ 

/ 


• • 
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Table *4 



Numbelc and Percentage of Incorrect Student Responses 
Matching Distracters Generated by Non-Systematic and Systematic Approaches 





NUMBER 


Annroach to 


rhP feneration of Response Option^ 




< 


INCORRECT 


SnhiP.rr Marten ExDertise 


Algorithmic Approach 


w 








N 


% 


ITEM ' 


RESPONSES 


> N 


% 


1 




4 


33% 


7 


58% 




i ^ 3 
1 ft 


1 
* 

2 


4% 
11% 


6 
9 


24% 


4 


1 7 


5 


29% 


7 


41% 


c 


52 


0 




4 


8% 


0 


±0 


1 


8% 


2 


15% 


7 


JO ^ 


2 


8% 


5 


19% 


Q 
O 


7ft 


19 


50%^, * 


17. f - 


45%- 




0 J. 


' ;2 


39% 


12 


39% 


10 


31' 


14 


45% 


- t 13, 


42% 


11 


20 


,3 


15% 


1 


. % 5% 




27 


. 5 


19% , 

> 


11 


41% 


13 


69 


3 ' 


4% 


8 


12% 


14 - 


» 61 /\ 


. g 


15% 


9 


15% 


15 


^,,71 


5 \- 


7% 


7 


10% 


' 16 


34 


8 


24% 


10 


29% 


' 17 


? 56 


1 


2% 


18- 1 


32% 


18 


56 


2 


4% 


4 


^7% * 


19 


72 




17% 


7 


10% 


• 20 


92 


1 


1% 




5% , 


TOTAL 


821 


109 


13% 


162 v 


20% 
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Table 5 

Number and Percentage -of Different Responses 
Given by Examinees Responding Incorrectly to Completion Items 

N»131 



Item 



N Different 
8 
21 
i4 * 
13 
38 
11 
19 
23 
20 

* V 17 
18 
18 

45' " 

r- - 56 

{ " • 

N 29* \ 
22 
43 
44 

527 



N Incorrect 


% 


N 'Answering 


12 


67 


12 


25 


84 


25 


18 


78 


18 


17 


76 


16 


52 


73 * 


49 


J3 


85 


11 


26 


73 


. 25 


38 


61 


37 








31 


65 


28 


- 31 


55 . 


' 29 


20 


90 




27 


67 




69 


65 


* 65 . . 


61 


82 


55 




79 


63 


34 


53 


25 > 


56 


52 


44 

* 


56 


39 


24 


4 72 • 


60 


49 


^ 92 ' 


48 


53 



% 



1 

2 
3 
4 
5 
6 
7 
8 

9 * 
10 
11 
12 
13 
14 
15 

* 16 

li- 
lt 18 4 ^ 
19 • 

20 

TOTAL 



821 



64 



672 



, 67 
' 84 
• .78 

.81 
-78 
100 
-76 
62 
. 71 
59 
95 
* 72 
69 
'91 
89 
72 
66 
92 
88 
\83 

> 78 



